📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:236 📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:165-176 📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:207-209
Python's multiprocessing forbids daemon processes from creating child processes — doing so raises AssertionError: daemonic processes are not allowed to have children.

ScanDagsTask.run() delegates to DagFileProcessorManager (Airflow 3.0+), SchedulerJobRunner (Airflow 2.6+), or SchedulerJob (older), all of which internally fork child processes to parse DAG files. Setting daemon=True on the ScanDagsTask process will cause it to crash immediately when it tries to spawn those children.

The singleton guard and join() logic are correct and sufficient to prevent zombies. The daemon=True flag should be removed.

📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:229-232
When a scan is already in progress and a new deploy triggers scan_dags_job_background(), the call is silently skipped (line 231-232). If the running scan started before the new DAG file was written to disk, the new DAG won't be picked up until the next manual deploy or scheduled scan.

Consider queuing a single follow-up scan (e.g., a boolean _rescan_requested flag checked after join) so that at most one additional scan runs after the current one completes. This ensures newly deployed DAGs are always picked up without spawning unbounded processes.

📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:214 📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:229-231 📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:238 📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:244
The _rescan_requested flag is set to True when a scan is already in progress (line 238) and cleared when a new scan starts (line 244), but nothing ever reads the flag to actually trigger a follow-up scan. The docstring (lines 229-231) promises that "the request is deferred ... so newly deployed DAGs are picked up once the current scan finishes," but this never happens — the flag is write-only.

This means the original "silently skipping" problem from the previous review is not actually fixed; it's just given a flag name. A deploy that arrives while a scan is running will still be lost.

To actually honour the flag, after the current scan finishes (detected at lines 236-242), check _rescan_requested and start a new scan if it's True. For example:

with _scan_lock:
    if _current_scan is not None:
        if _current_scan.is_alive():
            _rescan_requested = True
            logger.info("DAG scan already in progress, queued rescan")
            return
        _current_scan.join(timeout=5)
        _current_scan = None
        if not _rescan_requested:
            return  # no rescan needed, original caller already ran

    _rescan_requested = False
    process = ScanDagsTask()
    process.start()
    _current_scan = process

Alternatively, the docstring should be updated to stop claiming deferred rescans occur.

📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:236-244 📄 openmetadata-airflow-apis/tests/unit/test_scan_dags_singleton.py:73-87
When scan_dags_job_background() is called for a new deploy and finds a previously finished (not alive) scan process with _rescan_requested == False, it joins the old process but then returns at line 244 without starting a new scan for the current deploy.

Concrete scenario:

1. Deploy A → starts scan process, scan completes.
2. Deploy B → calls scan_dags_job_background(). _current_scan is not None (stale finished process), is_alive() returns False. Code joins it, sets _current_scan = None, checks _rescan_requested which is False → returns without scanning.
3. Deploy B's DAG is never picked up by the scheduler until some future deploy happens to trigger a scan.

Since scan_dags_job_background is only called on deploy (no periodic trigger), this means any deploy that arrives after the previous scan has finished will be silently dropped. This is the common case — most deploys don't overlap with a running scan.

The if not _rescan_requested: return guard at lines 243-244 conflates two situations: (a) a stale process being cleaned up with no new work needed (not a real scenario — the function is only called when there IS work to do), and (b) a new deploy request that should always trigger a scan.

The test test_no_new_scan_when_finished_without_rescan_flag asserts this broken behavior as correct.

📄 openmetadata-airflow-apis/openmetadata_managed_apis/api/utils.py:237-240
When a deploy arrives while a scan is running, _rescan_requested is set to True and the function returns. However, there is no callback, timer, or polling mechanism to trigger the deferred rescan once the current scan finishes. The rescan only happens if another deploy request arrives later and calls scan_dags_job_background() again.

Scenario:

1. Deploy A → starts scan.
2. Deploy B (while A's scan is alive) → sets _rescan_requested = True, returns.
3. A's scan finishes. No further deploys arrive.
4. Deploy B's DAG is never scanned.

This is a real concern because the whole point of the _rescan_requested flag is to handle rapid successive deploys, which by definition may not have a third deploy to trigger the queued rescan.

Consider adding a lightweight polling thread or a completion callback that checks _rescan_requested after the process finishes and re-invokes the scan if needed.